Skip to content

⚠️ Improve chained upgrade observability #12973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 10, 2025
Merged

⚠️ Improve chained upgrade observability #12973

merged 2 commits into from
Nov 10, 2025

Conversation

@fabriziopandini
Copy link
Member

@fabriziopandini fabriziopandini commented Nov 7, 2025 •
edited by sbueringer
Loading

What this PR does / why we need it:
Make sure all lifecycle hooks are logging in a consist way, and also makes the Topology reconcile condition to surface details about progress on the chained process.

Given the complexity of this process following reasons have been deprecated

	ClusterTopologyReconciledControlPlaneUpgradePendingReason = "ControlPlaneUpgradePending"
	ClusterTopologyReconciledMachineDeploymentsCreatePendingReason = "MachineDeploymentsCreatePending"
	ClusterTopologyReconciledMachineDeploymentsUpgradePendingReason = "MachineDeploymentsUpgradePending"
	ClusterTopologyReconciledMachinePoolsUpgradePendingReason = "MachinePoolsUpgradePending"
	ClusterTopologyReconciledMachinePoolsCreatePendingReason = "MachinePoolsCreatePending"
	ClusterTopologyReconciledMachinePoolsUpgradeDeferredReason = "MachinePoolsUpgradeDeferred"
	ClusterTopologyReconciledHookBlockingReason = "LifecycleHookBlocking"

e.g. ControlPlaneUpgradePending will remain true for most of the chained upgrades, thus making harder to follow progress.
Notably, this reason is used till the entire upgrade is completed, including lifecycle hooks.

As a replacement the following reason ha been introduced, and with a corresponding multiline message providing details on the upgrade process.

	// ClusterTopologyReconciledClusterUpgradingReason documents reconciliation of a Cluster topology
	// not yet completed because a cluster upgrade is still in progress.
	ClusterTopologyReconciledClusterUpgradingReason = "ClusterUpgrading"

Please also note that theLifecycleHookBlocking reason is not used anymore, because hooks are integral part of the cluster upgrade workflow (or of the cluster provisioning/deletion workflows).

Part of #12720

/area runtime-sdk

@k8s-ci-robot k8s-ci-robot added area/runtime-sdk Issues or PRs related to Runtime SDK cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 7, 2025
@fabriziopandini fabriziopandini force-pushed the improve-chained-upgrade-observability branch from 5b7cf3e to 1938697 Compare November 7, 2025 17:49
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 7, 2025
@fabriziopandini fabriziopandini force-pushed the improve-chained-upgrade-observability branch from 1938697 to c1db929 Compare November 7, 2025 18:16
@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 7, 2025

/test

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 7, 2025

@fabriziopandini: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test pull-cluster-api-build-main

/test pull-cluster-api-e2e-blocking-main

/test pull-cluster-api-e2e-conformance-ci-latest-main

/test pull-cluster-api-e2e-conformance-main

/test pull-cluster-api-e2e-latestk8s-main

/test pull-cluster-api-e2e-main

/test pull-cluster-api-e2e-main-gke

/test pull-cluster-api-e2e-mink8s-main

/test pull-cluster-api-e2e-upgrade-1-34-1-35-main

/test pull-cluster-api-test-main

/test pull-cluster-api-test-mink8s-main

/test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-apidiff-main

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-apidiff-main

pull-cluster-api-build-main

pull-cluster-api-e2e-blocking-main

pull-cluster-api-test-main

pull-cluster-api-verify-main

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 7, 2025

/test pull-cluster-api-e2e-main-gke

1 similar comment
@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 8, 2025

/test pull-cluster-api-e2e-main-gke

@fabriziopandini fabriziopandini force-pushed the improve-chained-upgrade-observability branch from c1db929 to d5b0a2d Compare November 9, 2025 14:03
@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 9, 2025

/test pull-cluster-api-e2e-main-gke

@fabriziopandini fabriziopandini force-pushed the improve-chained-upgrade-observability branch from d5b0a2d to 67cc286 Compare November 9, 2025 14:11
@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 9, 2025

/test pull-cluster-api-e2e-main-gke

1 similar comment
@sbueringer
Copy link
Member

sbueringer commented Nov 10, 2025

/test pull-cluster-api-e2e-main-gke

@sbueringer
Copy link
Member

sbueringer commented Nov 10, 2025

/test pull-cluster-api-e2e-main-gke

Just for more data :)

@sbueringer sbueringer mentioned this pull request Nov 10, 2025
Open
36 tasks
@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 10, 2025

I have looked into the autoscaler test error, and IMO it is un unrelated test flake. I have documented my finding in #12298 (comment)

sbueringer
sbueringer reviewed Nov 10, 2025
View reviewed changes
@fabriziopandini fabriziopandini force-pushed the improve-chained-upgrade-observability branch from 28115c9 to 8a124da Compare November 10, 2025 15:42
@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 10, 2025

/test pull-cluster-api-e2e-main-gke

@sbueringer
Copy link
Member

sbueringer commented Nov 10, 2025

Thank you very much!!

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 10, 2025
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 10, 2025

LGTM label has been added.

Git tree hash: 6873661b415c4f9a54e0cee123d851a306147ec0

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 10, 2025
@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 10, 2025

/test pull-cluster-api-e2e-blocking-main

@fabriziopandini fabriziopandini added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Nov 10, 2025
@fabriziopandini fabriziopandini force-pushed the improve-chained-upgrade-observability branch from 8a124da to e2f3b0c Compare November 10, 2025 17:38
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 10, 2025
@fabriziopandini
Copy link
Member Author

fabriziopandini commented Nov 10, 2025

/test pull-cluster-api-e2e-main-gke

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 10, 2025 •
edited
Loading

@fabriziopandini: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cluster-api-apidiff-main e2f3b0c link false /test pull-cluster-api-apidiff-main

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@sbueringer
Copy link
Member

sbueringer commented Nov 10, 2025

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 10, 2025
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 10, 2025

LGTM label has been added.

Git tree hash: 06ce90b4f9ae5a9909fcdfd078361c233ee5f571

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 10, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 9ea1f58 into kubernetes-sigs:main Nov 10, 2025
17 of 18 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.12 milestone Nov 10, 2025
@fabriziopandini fabriziopandini deleted the improve-chained-upgrade-observability branch November 11, 2025 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@richardcase richardcase Awaiting requested review from richardcase

@sbueringer sbueringer Awaiting requested review from sbueringer

Assignees

@sbueringer sbueringer

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/runtime-sdk Issues or PRs related to Runtime SDK cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.

Projects

None yet

Milestone

v1.12

Development

Successfully merging this pull request may close these issues.

3 participants

@fabriziopandini @k8s-ci-robot @sbueringer